Aiming at the problem that object segmentation algorithms based on single color information are very sensitive to the changes on lighting, a novel approach to detect target based on the fusion of color and depth information was proposed. Firstly, improved Visual Background Extractor (ViBe) and multiple-frame subtraction algorithm were used to establish models for RGB and depth images which captured by Kinect senor respectively. Then, strategy of Selection Criterion (SC) was used to optimize segmentation results. Lastly, most likely target was labeled by calculating similar degree between foreground and template in the rg chromaticity space. The experimental results demonstrate that the proposed method exhibit a certain degree of resilience to light disturbance and noise, and it can overcome the disadvantages of single RGB based algorithms effectively.